Repository navigation
Count tokens with the Rust tokenizers crate on wasm32 - #2
Merged
Merged
Conversation
Replace @huggingface/tokenizers with Hugging Face tokenizers 0.23 compiled to wasm32-unknown-unknown. The vocab is embedded, fancy-regex is used instead of onig, and counts stay the same. Unpaired surrogates are rejected with 400 because a Rust string cannot hold them. Co-authored-by: Jowell Young <guza@duck.com>
format cargo code
The parity helper was only a local id comparison tool. Nothing in the wasm crate reads JSON, so the direct serde_json dependency goes with it. Co-authored-by: Jowell Young <guza@duck.com>
Scripts, the bench, and CI run through pnpm and Node. The bench builds the wasm module first and exits if it does not finish within 60 seconds. Co-authored-by: Jowell Young <guza@duck.com>
An 8 KiB digit run, the slowest input measured in workerd, stays around 5ms. The body cap covers a unicode-escaped prompt of that size. MAX_BODY_BYTES and MAX_PROMPT_BYTES override the defaults. Co-authored-by: Jowell Young <guza@duck.com>
A positive integer var replaces that default. The caps no longer check each other. Co-authored-by: Jowell Young <guza@duck.com>
Encode timing stays in the bench worker. Init time and wasm memory are no longer exported from the tokenizer. Co-authored-by: Jowell Young <guza@duck.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces
@huggingface/tokenizerswith the Hugging Facetokenizers0.23 crate compiled towasm32-unknown-unknown. The built.wasmis not committed.scripts/build-wasm.shproduces it, andpnpm test,pnpm check,pnpm dev, andpnpm deployrun that script first. The package manager is pnpm 12. CI installs Node, pnpm, the Rust toolchain, and thewasm32-unknown-unknowntarget, then runspnpm install --frozen-lockfile,pnpm check, andpnpm test. This does not deploy.Free-plan caps are now 8 KiB of rendered prompt and 48 KiB of raw body, sized so a worst-case digit run stays around 5 ms under the 10 ms CPU limit.
MAX_PROMPT_BYTESandMAX_BODY_BYTESoverride those defaults. A paid-plan example (128 KiB prompt, 768 KiB body,cpu_ms = 200) is in the README.limits.cpu_msis not set inwrangler.toml, because the free plan rejects it.Choices
tokenizers0.23.2,default-features = false, featureunstable_wasm. That selectsfancy-regexand leaves out the Conigandesaxxlibraries.config/deepseek-v4/tokenizer.jsonis embedded in the wasm module withinclude_bytes. Encode usesencode_fast(same ids, no offset tracking) and does not add special tokens. The chat template still writes those tokens into the prompt.ahashasksgetrandomfor map seeds while the vocab is built. Workerd rejectscrypto.getRandomValueson a wasm-memory view (the stockwasm_jsbackend), so the wasm build uses a custom getrandom backend that copies through a JavaScriptUint8Array.invalid_request_error. A Rust string cannot hold them. The previous JavaScript tokenizer counted U+D800 as one id. Well-formed text, including emoji, is unchanged.encode_fastids matchedencodeids.Workerd before and after
Same harness (
bench/workerd.mjs), wrangler 4.149, compatibility date 2026-10-09. Encode time is the median of 5 warm requests, each averaging 30 encodes inside the isolate.performance.nowduring module startup does not move in this workerd, so cold start is the first request's wall clock (isolate startup plus one tiny encode plus the local proxy). The JavaScript column is one run. The wasm column is three runs.@huggingface/tokenizerstokenizers0.23.2 wasmThe API worker dry-run is 8732 KiB uncompressed / 2676 KiB gzip. Both sit under the 64 MiB uncompressed script limit. Wasm memory after init is 57.7 MiB, under the 128 MiB isolate limit. The cold request is under the 1 second startup CPU budget; most of that wall time is outside the encode itself.
Counts on the existing suite match, including the README prompt at 11 tokens.